The present invention relates to an electronic camera and an image processing method, more specifically, in a single plate electronic camera employing a photoelectric conversion element and a color filter, the electronic camera wherein a color signal value of which color is different from a filter color of each pixel is obtained by interpolation, and a method of an image compression and decompression process employed in said electronic camera and the like.
Conventionally, the single board sensor electronic camera has been known wherein a color filter is put on each of pixels of a photoelectric conversion element such as a CCD (Charge Coupled Device), and color image signals are obtained.
In the above-mentioned single board sensor electronic camera, as shown, for example, in FIG. 2, a mosaic color filter is employed wherein in order to obtain a brightness signal (Green signal) for which a high degree of resolution is required, Green filters are arranged in a checkerboard pattern, while in order to obtain two kinds of color signals (Red and Blue signals), Red filters and Blue filters are arranged in the checkerboard pattern.
Furthermore, as the above-mentioned color filters, there are one composed of R, G, B as shown in FIG. 2 and the other composed of the combination such as (W, G, Cy, Ye), (G, Cy, Ye) and (Mg, G, Cy, Ye), etc. wherein W represents white; Cy represents cyan; Ye represents yellow and Mg represents magenta.
In the single board sensor electronic camera as mentioned above, when employing, for example, color filters consisting of R, B, G, information on any one of R, G, B for each pixel is only obtained. Accordingly, there is a case such that the interpolation calculation on image signals is performed and each of the R, G, B data is obtained for each pixel.
For example, U.S. Pat. No. 4,642,678 discloses a structure wherein in an arrangement utilizing an R, X, B mosaic filter, when a Green signal value is obtained by the interpolation for the pixel of an s filter or a B filter, the average value of four G filter pixels adjacent to a target pixel is put as a Green signal of the target pixel. Furthermore, there is a disclosure of a structure wherein when a Red signal value and Blue signal value are obtained by interpolation, the Red signal value and the Blue signal value are linearly interpolated according to the signal value of a pixel of a Red filter and a Blue filter adjacent to the target pixel and a Green signal value interpolated at said adjacent pixel, the Green signal value interpolated at said adjacent pixel and the Green signal (original signal value for G filter pixel and interpolated value for the R and B filter pixels) at the target pixel.
Furthermore, xe2x80x9cDigital Camera Utilizing Newly Developed Compression and Interpolation Processingxe2x80x9d in the Proceedings of Fine Imaging Symposium (1995) of the Japan Photographic Society discloses a structure wherein upon recognizing an edge pattern, an interpolation direction is set which is pertinent for the recognized pattern, and the interpolation is performed using a pixel signal value in said interpolation direction.
Further, U.S. Pat. No. 5,373,322 discloses a structure wherein in an arrangement composed of a R, G, B mosaic filter as shown in FIG. 2, for the interpolation, for example, of a Green signal, the gradient of a color signal Blue and Red for an target pixel is obtained and the direction suitable for the interpolation is determined according to said gradient, and the interpolation value is obtained.
In the following, is shown examples of the interpolation calculation of G signal 34 at pixel R34 of a Red filter shown in FIG. 2 and a G signal G43 at a pixel B43 of a B filter.
In the interpolation operation of G34, at first, the gradient for R34 from R32, R36 and R14 is calculated according to the following expressions.
Hdiff=|(R32+R36)/2xe2x88x92R34|
Vdiff=|(R14+R54)/2xe2x88x92R34|
wherein Hdiff represents the gradient in the horizontal direction against R34 and Vdiff represents the gradient in the perpendicular direction.
And, when Hdiff less than Vdiff,
put G34=(G33+G35)/2.
When Hdiff greater than Vdiff
put G34=(G24+G44)/2.
When Hdiff=Vdiff
put G34=(G24+G44+G33+G35)/4.
In the same way, in the interpolation operation of G43, the gradient for B43 is first calculated from B41, B45, B23 and B63 according to the following expressions.
Hdiff=|(B41+B45)/2xe2x88x92B43|
xe2x80x83Vdiff=|(B23+B63)/2xe2x88x92B43|
And, when Hdiff less than Vdiff,
put G43=(G42+G44)/2.
When Hdiff greater than Vdiff,
put G43=(G33+G53)/2.
When Hdiff=Vdiff,
put G43=(G33+G53+G42+G44)/4.
On the other hand, the interpolation of R signals and B signals is performed under such a structure that a linear interpolation is carried out utilizing a pixel of a G filter and signals of a R filter and a B filter adjacent closely to said pixel.
In the following, the interpolation operation expressions are shown for R33, R43 and R44 of the R signal.
R33=((R32xe2x88x92G32)+(R34xe2x88x92G34)/2+G33
R43=((R32xe2x88x92G32)+(R34xe2x88x92G34)+(R52xe2x88x92G52)+(R54xe2x88x92G54)/4+G43
R44=((R34xe2x88x92G34)+(R54xe2x88x92G54)/2+G44
In addition, in the following, are shown operation expressions of B33, B34 and B44 of the B signal.
B33=((R23xe2x88x92G23)+(R43xe2x88x92G43)/2+G33
B34=((R23xe2x88x92G23)+(R25xe2x88x92G25)+(B43xe2x88x92G43)+(R45xe2x88x92G45))/4+G34
xe2x80x83B44 =((B43 xe2x88x92G43)+(B45 xe2x88x92G45))/2 +G44
On the other hand, in the above-mentioned electronic camera, conventionally, the image data have undergone compression which are recorded in a semiconductor memory, etc. and for said compression, a method utilizing an orthogonal transformation coding has been mainly employed.
For example, in the JPEG (Joint Photographic Coding Experts) compression, the RBG signals are subjected to orthogonal transformation through DTC (Discrete Cosine Transformation) while putting 8xc3x978 pixel as one unit, and then quantized, and Huffman coded to be compressed data. The compressed data are stored or transmitted. When the compressed data are decompressed (extended), the image is reproduced thorough the reverse process mentioned above.
As mentioned above, the orthogonal transformation coding is performed under dividing the image region into a plurality of blocks. Therefore, in the image decompression (image extension), there has been a problem such as a phenomenon wherein the joint of the blocks is not natural (hereinafter referred to as block deformation). various methods have been proposed for improving said block deformation.
For example, Japanese Patent Publication Open to Public Inspection No. 63-236088 discloses a structure wherein the orthogonal transformation coding is performed so that each block is overlapped, and Japanese Patent Publication Open to Public Inspection No. 3-166825 discloses a structure wherein a low-pass filter is applied to the portion which is judged to be a flat portion of an image. Furthermore, Japanese Patent Publication Open to Public Inspection No. 4-2273 discloses a structure wherein random noise is added to the neighbor of a block boundary or a low-pass filter is applied to that. Still further, Japanese Patent Publication Open to Public Inspection No. 6-113147 discloses a structure wherein for an image having a block deformation, a low-pass filter is applied to the portion having the deformation upon judging the presence of the deformation from the boundary.
Incidentally, in the interpolation method in the mosaic filter disclosed in the above-mentioned U.S. Pat. No. 4642678, in the Blue filter pixel, when obtaining the Red signal value by interpolation, the Green and Red signals are only employed. In the same way, when interpolating the Blue signal by the Red filter pixel, the structure is such that the Green and Blue signal values are only employed. In the interpolation of the B signal in the R filter pixel, a structure is such that the signal value of the target pixel is not employed for the interpolation operation. Therefore, according to the above-mentioned interpolation method, problems are caused such that the color reproduction and sharpness are deteriorated.
In a method which determines the direction of interpolation operation by performing the pattern recognition of an edge, there have been a problem such that the operation load is large due to the discrimination of an image pattern and the interpolation process is slow. Furthermore, there have been another problem such that because in the same way as mentioned above, B and R signal values are interpolated according to signal values of G and B or G and R without utilizing the target signal value, the color reproduction is not sufficient.
Furthermore, in U.S. Pat. No. 5,373,322, there is a problem such that the interpolation of G signals which are brightness signals is performed according to the gradient of R and B signals which are color signals, and as a result, the interpolation is performed irrespective of the G gradient. Further, regarding to the interpolation of the color signals, there is another problem such that because the linear interpolation is performed utilizing the interpolated G signals, the color reproduction of a whole image is not sufficient. Furthermore, a defect is caused such that the process is slow because of judging the gradient.
Furthermore, in each of conventional technologies mentioned above, the interpolation is performed utilizing a pixel in one dimensional direction or pixels adjacent closely to the target pixel. In such the structure, there has been a problem such that the color reproduction is not sufficient because the number of reference pixels is small.
On the other hand, in the above-mentioned conventional methods for improving the block distortion, when letting the blocks overlap, there is a problem such that it is required to undergo process different from a standard compression method. In a structure wherein the random noise is added, there has been a problem such that an image becomes rough due to the addition of the noise. Furthermore, in a structure wherein the low-pass filter is applied upon judging the flat portion, or wherein the low-pass filter is applied upon judging the presence of the block distortion, the operation load is enlarged because the judgment process is required and when the judgment process is not performed suitably, on the contrary, the image quality tends to be degraded.
In view of the above-mentioned problems, the present invention has been accomplished. An object of the present invention is to provide, in a single board sensor electronic camera, an interpolation process which results in excellent color reproduction and sharpness, and further shortens processing time.
In addition, another object of the present invention is to provide a method wherein in a simple image processing, a block distortion is decreased without causing the deterioration of image quality.
The present invention which accomplishes the above-mentioned objects is an image processing method of an electronic camera which comprises a photoelectric element having plural sensors and color filters in plural different colors, wherein each of the plural sensors corresponds to a respective one of the color filters in the different colors. The image processing method comprises the following processes: 1) when one of the plural sensors is assigned to a target sensor, a process for obtaining a signal value from the target sensor, 2) in the desired region where the target sensor is positioned in the center, a process for obtaining the first average value from signal values of the plural sensors having the color filters of which a color is the same as that of the color filter corresponding to the target sensor, 3) in the desired region, a process for obtaining the second average value from signal values of the plural sensors having the color filters of which a color is different from that of the color filter of the target sensor and 4) according to the signal value of the target sensor, the first average value and the second average value, a process for obtaining an interpolation value for a color, which is different from that of the color filters corresponding to the target sensor, of the target sensor.